1. Field of the Invention
This invention relates to a proposal of a user interface environment adapted to one or more than one users operating objects relative to computers and inputting commands and data to computers. More particularly, the present invention relates to a proposal of a user interface environment where users can operate objects relative to computers and input commands and data to computers in a highly intuitive and easily understandable way.
Still more particularly, the present invention relates to a proposal of a user interface environment where operations interconnecting a plurality of computers are expressed in a highly intuitive and easily understandable way. Above all, the present invention relates to a proposal of a user interface environment in an information space produced by extending disk top display screen images native to computers to the real world to fuse digital spaces of the computers and the physical real, in which user interface environment operations of individual users and concerted operations of a plurality of users are expressed in a highly intuitive and easily understandable way.
2. Related Background Art
As a result of technological innovations in recent years, compact general purpose computer systems in the form of work stations and personal computers (PCs) with high added values and enhanced functional features have been developed and marketed at low prices. Such computer systems are almost ubiquitous in various research institutions including colleges and universities, commercial firms and offices and ordinary homes where people spend their daily lives.
One of the recent trends in the field of computer technology is down-sizing of computers. Particularly, compact and lightweight portable computers (mobile computers) have become very popular owing to the recent development of semiconductor technology that has made very small LSI (large scale integration) chips commercially available and boosted the processing capacity of computer processors.
Portable computers include so-called “notebook computers” and “PDAs (personal digital assistants)” that refer to palm top size computers. Notebook computers have a so-called cram-shell structure where a lid that carries an LCD (liquid crystal display) on the inner surface thereof is swingably linked by a hinge device to a computer main body that carries a keyboard on top.
Portable computers are adapted to operate in a mobile environment and drive one or more than one computer systems that may also be mobile. As a result of the advent of such lightweight portable computers, computers are almost ubiquitous nowadays in daily home lives as well as in business scenes where they take significant roles for supporting businesses. For instance, these days most people attend conferences and business presentations with their notebook computers.
There have been remarkable improvements in terms of computing resources including data processing capabilities of CPUs (central processing units) and hard disk capacities. They are by no means less power than desk-top computers. For instance, notebook computers are normally provided with a hard disk whose storage capacity (several to tens of several giga bits) is sufficient for storing all the documented data that a person normally possesses including the data on the presentations made by the possessor of the computer. Therefore, the person who are making a business presentation can draw out any necessary data from the hard disk so that he or she may distribute them to and share them with the attendants of the presentation and exchange them with date of some of the attendants. Thus, it is no longer necessary to copy part of the data stored in the hard disk to a floppy disk or vice versa.
Most computer systems provide the user with an interactive processing environment so that the computer system operates in response to the command issued to it by the user and the outcome of the processing operation is displayed on the display screen of the system. Currently, there is occurring a shift from the character-based user input environment or “the CIU (character user interface) environment” such as DOS (disk operating system) shell screen to the graphic-based user input environment or “the GIU (graphic user interface) environment”. In the GUI environment, a disk top unit simulating a computer system and numerous icons are provided for the display screen.
In the case of a disk top type computer provided with a GUI, all resource objects to be handled in the computer system including files are expressed by icons. The user can intuitively issue commands by directly operating icons on the display screen that represent respectively programs, data, holders and devices by means of a mouse and other I/O devices (with motions such as click, drag and drop, etc.). Therefore, the user is no longer required to memorize specific commands and learn how to handle them. Nor he or she is required to learn how to operate the keys of the keyboard for issuing commands, which is normally a very cumbersome task.
Another of the recent trends in the field of computer technology is interconnection. Computers may be interconnected typically by way of local interfaces such as serial ports and parallel ports or by way of network interface cards (NICs) so that they may be connected to each other by way of one or more than one telecommunication networks. From the viewpoint of interconnection of computers, telecommunication networks include LANs (local area networks) provided on single sites, WANs (wide area networks) realized by connecting LANs by way of dedicated lines and the internet which is a global telecommunication network realized by connecting various networks.
Interconnection of computers provides advantages including that information and computer resources are shared by a number of computer users, that information is delivered and disseminated highly efficiently and that a plurality of computers can be driven in a cooperative and concerted manner to carry out a common mission. For instance, an operation of processing a digital object can be carried out highly efficiently when a plurality of computers are made to cooperate in an intensive way.
Particularly, in recent years, the user can link his or her computer easily to a telecommunication network as a result of the advent of proximity telecommunication systems such as wireless LAN that is a wireless version of Ethernet and blue tooth [1]. The computers and peripheral devices (e.g., printers) installed in a building can mostly be connected to a telecommunication network so that they may be shared for operation by a plurality of users. Additionally, computer users can carry their own computers to any place they want and link them to a telecommunication network without difficulty.
In a distributed computing environment, the computers connected to a network are in a “transparent” state and they can be operated by means of an application program regardless if it is stored in a local storage disk or in a remote disk. In other words, the user is no longer required to know the locations of resource objects such as programs and data because they can be exchanged seamlessly among computers.
The user can link his or her computer to a telecommunication network without any difficulty. However, if the user wants to specify a particular computer or peripheral device (target) to which data are to be transferred by way of the network, he or she has to know its ID (or its address) if it is located just in front of the user. In other words, even in a transparent and distributed computing environment, user operations are rather indirect and the related computers are not cooperative so that the user cannot act intuitively.
For example, if the user want to transfer an object between two or more than two computers in a currently available GUI environment, he or she is required to display a list of the devices linked to the network on an explorer screen and find out the icons simulating the respective computers before proceeding to a “drag and drop” operation.
A technique for solving the user operation problem may be the use of a real world-oriented user interface, which represents the technology for expanding the information space that is defined by the scope of the user operation from the “digital space” produced on a display screen of a computer to the space of the real world in a continues way. With such a real world-oriented interface, the display screen of the computer may be expanded to the surface of the disk on which the computer is placed and also to the surface of the walls of the room where the computer is located. In other words, the information space of the user expands beyond the limit of the digital space of the stand-alone computer to the entire room so that digital objects of the digital space of the computer and physical objects of the real world can be handled in an integrated manner.
For instance, “pick-and-drop[2]” is an expansion of user operation “drag-and drop” that is reliably being utilized in the GUI environment and represents the technology of advanced interactions for mapping the transfer of objects such as data among a plurality of computers into an intuitive motion in the real world.
With the “pick-and-drop” technique, seamless “pick-and-drop” operations can be realized as a result of cooperation of a plurality of disk top computers. For example, when an object is dragged by operating the user input device that may be a track ball or a stick of a disk top computer to move the pointer in the display screen beyond an edge thereof, the drag operation may be succeeded to the display screen of the adjacently located disk top computer so that the pointer of the latter display screen appears, dragging the object.
On the other hand, the pointer in the display screen of the former disk top computer disappears, while dragging the object. This operation of the pointer of the former disk top computer picking up an object (digital object) is referred to as “pick” while the operation of causing the pointer of the latter disk top computer to appear in order to continuously drag the object is referred to as “drop”. Thus, the entity of the object (e.g., a data file) is transferred behind the screen between the two computers by way of the network. In other words, the succeeded drag operation accompanies a movement of the entity of the object. FIG. 1 schematically illustrate an “pick-and-drop” operation of an digital object of adjacently located computers.
As for “pick-and-drop” operations, refer to Japanese Patent Application Laid-Open Publication No. 11-53236 whose assignee is the applicant of the present patent application, specifically passages [0052] through [0059] and FIGS. 6 through 9 of the above patent document.
On the other hand, the Japanese Patent Application Laid-Open Publication No. 11-316461 that has been assigned to the applicant of the present patent application describes an information input/output system that expands the user's work space, or the disk top, that may be represented by a GUI into the real world and provides a real-world-oriented user interface with which the user can issue commands to the computer in an intuitive and ease-to-understand fashion.
The information input/output system as described in the above patent document can provide an information space, which is a user's work space, by dynamically combining an environment type computer group arranged in the real space including computers of the table type and those of the wall type and a portable type computer group including computers that users can carry with.
A camera-based object recognition system may be used for such an information input/output system. Then, an image of a computer display screen may be projected on a wall surface by means of a projector. Then, both the projected image, which is an object in the local space of the computer, and an object in the real world (which, for example, exists on a table) can be recognized on the basis of the image picked up by the camera and the movements of the objects can be traced.
In such an information environment, tables and walls in a conference room or a presentation room are computerized and may three-dimensionally expand a portable computer to provide a large work space (an expanded desk top display) to the user. Then, objects (such as documents, cards, vide tapes) in the real world may be identified typically by means of “marker recognition” so that information may be exchanged between them.
Additionally, the user can easily incorporate the portable computer he or she brings in into an information space where a number of computers are installed so that the portable computer may be made to cooperate with the other computers in a concerted manner.
Thus, the user can handle information by three-dimensionally making use of the positional relationship of various objects in the real world without paying attention to the addresses and the IDs of the individual objects including devices by using direct handling techniques that are not limited to the closed work spaces of single computers.
In an information space as disclosed in the above patent document, the user can use a technique of “hyperdragging”[3], with which he or she continue an operation of dragging an object in the digital space of a computer display in the real world. For example, the user can continue an operation of dragging a cursor by means of a mouse on the surface where the computer display is placed (see FIG. 2) and then further to a wall of the room (see FIG. 3). In other words, the entire room where the computer is placed can be used as part of the computer display to handle objects.
With the technique of “hyperdragging”, the user can interactively operate an object only by intuitively recognizing the physical position in the real world that may include the positional relationship between the object and the computer (e.g., “the object is located at a side of the computer”). Additionally, while the user is operating an object at the front end, processing operations may be proceeding in the background among the related computers (e.g., network transfer operations) to move entities of digital objects such as document files.
Furthermore, with the technique of “hyperdragging”, it is possible for the user to move data in the real world simply if he or she knows how to use a mouse (“drag and drop”) and the physical location of the target device (e.g., “the device on the table”). For example, when moving and exchanging digital objects between two adjacently located portable computers by using the “hyperdragging” technique (see FIG. 4), the user does not need to know how to set up a network environment to connect the computers nor the address of the target device.
With an information input/output system as disclosed in the above patent document, a physical real space can be linked to a logical space (digital space) in a computer to establish a relationship between a computer data and a physical object. For instance, a document holder may be attached to a physical object such as a VCR tape cartridge placed on a table (see FIGS. 14 through 14 of the above patent document).
Additionally, computer peripheral devices such as printers, displays and scanners can be made to support the “hyperdragging” feature. For instance, an operation of printing out a digital object can be started by hyperdragging and dropping it onto a printer that is installed in an information space.
It will become possible to issue user commands to a computer not only by way of native user input devices such as a mouse and/or a keyboard but also by way of various physical objects located dispersedly in the real world if a physical real space and a digital space in a computer can be fused further. It will also become possible to draw out a digital object by way of a physical object. Then, operating computers will become much more intuitive, easily understandable and attractive to the user.